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The vision of free expression that characterized much of the twentieth century is inadequate to protect free 
expression today. 


The twentieth century featured a dyadic or dualist model of speech regulation with two basic kinds of 
players: territorial governments on the one hand, and speakers on the other. The twenty-first-century model 
is pluralist, with multiple players. It is easiest to think of it as a triangle. On one corner are nation-states and 
the European Union. On the second corner are privately owned internet-infrastructure companies, 
including social media companies, search engines, broadband providers, and electronic payment systems. 
On the third corner are many different kinds of speakers, legacy media, civil-society organizations, hackers, 
and trolls. 


The practical ability to speak in the digital world emerges from the struggle for power between these various 
forces, with “old-school,” “new-school,” and private regulation directed at speakers, and both nation-states 
and civil-society organizations pressuring infrastructure owners to regulate speech. 


This configuration creates three problems. First, nation-states try to pressure digital companies through 
new-school speech regulation, creating problems of collateral censorship and digital prior restraint. Second, 
social media companies create complex systems of private governance and private bureaucracy that govern 
end users arbitrarily and without due process and transparency. Third, end users are vulnerable to digital 
surveillance and manipulation. 


This Essay describes how nation-states should and should not regulate the digital infrastructure consistent 
with the values of freedom of speech and press. Different models of regulation are appropriate for different 
parts of the digital infrastructure: Basic internet services should be open to all, while social media 
companies should be treated as information fiduciaries toward their end users. Governments can 
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implement all of these reforms—properly designed—consistent with constitutional guarantees of free 
speech and free press. 


* Knight Professor of Constitutional Law and the First Amendment, Yale Law School. Many thanks to Jameel Jaffer, 
Daphne Keller, Maggie McKinley, David Pozen, and Tim Wu for their comments on a previous draft. 
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Free speech is a triangle. The conception of free expression—and of the dangers to free expression 
—that characterized much of the nineteenth and twentieth centuries concerned whether nation- 
states and their political subdivisions would censor or regulate the speech of people living within 
their borders. That picture still describes many important free speech problems, yet it is increasingly 
outmoded and inadequate to protect free expression today. In the early twenty-first century, 
freedom of speech increasingly depends on a third group of players: a privately owned 
infrastructure of digital communication composed of firms that support and govern the digital 
public sphere that people use to communicate. 


Consider a few recent speech controversies. The first is the European Union’s “right to be 
forgotten.” It requires search engine companies (essentially Google) to eliminate certain newspaper 
articles from their search results. 1 A second is the recently passed German law known as NetzDG. 

2 It requires social media companies to take down many different kinds of speech, including hate 
speech, within twenty-four hours of a complaint. 3 A third is the concern about fake news propa- 
gating through social media sites. 4 A fourth is the decision by various internet companies— 
following the Charlottesville march in August 2017—to block, censor, or otherwise refuse to do 
business with various neo-Nazi and hate sites. 5 Each of these controversies concerns the new 
structure of speech regulation in the digital age. 


The twentieth century featured a dualist or dyadic system of speech regulation. © In the dualist 
model, there are essentially two players: the nation-state on the one hand and the speaker on the 
other. Nation-states regulated many different kinds of speakers and mass media of all kinds, 


including publishing houses, movie houses, newspapers, radio stations, and television stations. 


The twenty-first-century model is pluralist, with many different players. 7 For ease of exposition, 
we might consider it as a triangle. 


FIGURE 1: THE PLURALIST MODEL OF SPEECH REGULATION 8 
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On one corner of the triangle are nation-states, states, municipalities, and supranational 
organizations like the European Union. 


On the second corner of the triangle are internet-infrastructure companies. These include social 
media companies, search engines, internet service providers (ISPs), web-hosting services, Domain 
Name System (DNS) registrars and registries, cyber-defense and caching services (such as 
Cloudflare and Akamai), and payment systems (such as PayPal, Mastercard, and Visa). Each of 
these elements of the internet infrastructure is important, if not crucial, to people’s practical ability 
to speak. In most countries, this internet infrastructure, or important parts of it, are privately 


owned. 9 


On the third corner of the triangle, at the very bottom, we have speakers and legacy media, 
including mass-media organizations, protesters, civil-society organizations, hackers, and trolls. 
Although both states and infrastructure owners regulate their speech, they are sometimes able to 


influence states and infrastructure owners through social activism and protest. 1° 


Nation-states regulate speakers and legacy mass media through old-school speech regulation. 
Nation-states regulate and attempt to co-opt and coerce internet infrastructure through new- 
school speech regulation. Finally, the internet infrastructure regulates private speakers and legacy 
media through techniques of private governance. 
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This is the new structure of speech regulation in the early twenty-first century, and debates about 
the rights of online free expression must grapple with that structure. To understand how this new 
system works, we must understand the distinction between old- and new-school speech regulation, 
explained in Part |, and the emerging system of private governance, discussed in Part II. Parts II] and 
IV offer proposals for protecting freedom of speech in the changed environment. A brief conclusion 
follows. 


I. OLD-SCHOOL AND NEW-SCHOOL SPEECH REGULATION 


In traditional or old-school speech regulation, nation-states use threats of fines, penalties, 
imprisonment, or other forms of punishment or retribution to regulate or control the speech of 
individuals, associations, and media companies. ™ As noted above, this conception is dyadic. In 
this traditional conception, freedom of speech and press simply means being free of old-school 
speech regulation. 


Old-school speech regulation still exists around the world. But digital free speech has created new 
problems for which old-school methods are inadequate. The early twenty-first century has 


|? 


developed new methods for controlling digital speech. This is the “new school” of speech 
regulation, and because of its ascension, freedom of speech today requires far more than freedom 


from old-school speech regulation. 


Whereas old-school regulation is directed at speakers, new-school speech regulation is directed at 
the internet infrastructure. '2 Nation-states (or supranational entities like the European Union) 
attempt to regulate, threaten, coerce, or co-opt elements of the internet infrastructure in order to 
get the infrastructure to surveil, police, and control speakers. 13 In essence, nation-states attempt 
to get the privately owned infrastructure to do their work for them. 


Consider the free speech controversies mentioned in the Introduction. 14 Germany’s NetzDG is 
aimed at search engines and social media companies to limit forbidden speech. > The European 
Union’s “right to be forgotten” is directed (in part) at search engines in order to make it hard for 
people to discover embarrassing stories in newspapers. Calls for government regulation to prevent 
fake news demand that social media companies—and other parts of the internet infrastructure— 
take steps to limit the publication and distribution of false stories among end users. Following the 
Charlottesville protests, neo-Nazi sites were hampered or blocked not by states and municipal 


governments but by private-infrastructure owners. 


Although nation-states continue to regulate speech directly through old-school methods, they 
increasingly depend on new-school speech regulation—attempting to coerce or co-opt private 
owners of digital infrastructure to regulate the speech of private actors. For this reason, new-school 
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speech regulation affects the practical ability to speak every bit as much as old-school speech 


regulation. 


A. COLLATERAL CENSORSHIP AND DIGITAL PRIOR RESTRAINT 


New-school speech regulation poses two central problems for freedom of speech. First, it usually 
involves some form of collateral censorship. Second, it raises many of the same problems as prior 
restraint, except that the restraint is performed by private bureaucrats and algorithms in the service 
of the state. 


1. Collateral Censorship. — Collateral censorship occurs when the state targets entity A to control 
the speech of another entity, B. 16 The state tells A: Locate and block or censor B, or else we will 
punish or fine you. In effect, collateral censorship attempts to harness a private organization to 
regulate speech on the state’s behalf. 


Collateral censorship does not raise special problems for freedom of expression when A and B are 
part of the same entity or firm that produces the expression (for example, when B is A’s employee), 
or when A has a traditional editorial or publishing relationship to B. Defamation law holds 
newspapers liable for what their reporters write and their advertisers advertise, and it holds book 
publishers liable for their authors’ defamation. 17 When A is B’s book publisher, when B works for 
A, or when B advertises in A’s newspaper, the law assumes that A has a vested interest in defending 
and protecting the speech produced by B that A edits and publishes. 


We cannot make the same assumption, however, when A is part of the internet infrastructure and B 
is one of the countless number of people who use A’s services to communicate with others. Then A 
and B are not in the same relationship as the newspaper and its reporters, the publishing house and 
its authors, or the magazine and its advertisers. 


In these cases, A’s incentives are somewhat different. Told by the state that it must censor or block 
speakers like B, A will err on the side of caution. 18 It will tend to overblock or overfilter content, 
discarding the wheat with the chaff. In addition, A will be more likely to take down speech that 
anyone objects to or that it fears someone might object to. Because there are so many speakers 
(and because A wants to make the vast majority of its end users feel comfortable), denying access 
to a very small number of speakers will not damage A’s business model, whereas repeated 
imposition of government liability for the speech of total strangers might seriously hinder its ability 
to do business. 


2. Digital Prior Restraint. — Imposing liability on infrastructure providers unless they surveil and 
block speech, or remove speech that others complain about, has many features of a prior restraint, 


although technically it is not identical to a classic prior restraint. 19 


Administrative prior restraints deny people the right to speak without a full judicial determination of 
whether their speech is protected or unprotected and without the procedural protections of the Bill 
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of Rights. 2° In addition, administrative prior restraints place the burden of inertia on the speaker 
and the benefit of inertia on the government. ?! People have to get someone else’s permission 
before they can speak (or speak again, if the order comes down after they have begun to publish 
copies). Administrative review acts as a bottleneck to free speech; nothing will happen until the 
bureaucrat gets around to deciding, and the decision, when it occurs, may happen in secret with no 
transparency or due process. 27 


Many of these problems also occur when internet-infrastructure companies block, filter, or take 
down content. If end users are blocked, or their speech is taken down, they do not get to speak until 
somebody in the infrastructure company decides that they have permission. This blocking or 
removal occurs without any judicial determination of whether their speech is protected or 
unprotected, without any Bill of Rights protections, without any due process rights to a hearing 
before the action is taken, or indeed, without any obligation to consider and resolve end-user 
objections promptly. 233 Rather, some company functionary or bureaucrat—or algorithm—decides 
whether and when they get to speak. 74 


In this way, our twenty-first-century digital world has recreated the prior restraints of the sixteenth 
and seventeenth centuries, offering a twenty-first-century version of administrative prior restraint. 
There are two important differences, of course. First, although some content is blocked at the 
outset, other content is removed after appearing for a brief period of time. 25 Second, the restraint 
is not at the hands of government bureaucrats, but at the hands of privately owned companies who 
act to avoid threats of liability by nation-states. 26 


B. PUBLIC-PRIVATE COOPERATION AND CO-OPTATION 


This leads to the next key feature of new-school speech regulation: public-private cooperation and 
co-optation. Governments attempt to coax, cajole, or coerce private-infrastructure owners to do 
their bidding and to help them surveil and regulate speech. ?7 Public—private cooperation—or co- 
optation—is a natural consequence of new-school speech regulation. 


First, the technical capacities of infrastructure owners for identifying and removing content far 
outstrip those of most countries; hence it is easier to get private companies to perform these tasks 
for the government. 28 


Second, new-school speech regulation often depends on data surveillance—or else is in aid of data 
surveillance—because many methods of speech regulation require some ability to know what end 
users are doing. 2? Owners of private infrastructure are essential to effective data collection and 
surveillance; indeed, the very same infrastructure that makes broad participation in free expression 
possible is also the infrastructure that facilitates widespread digital surveillance. 3° 


Third, complementary incentives drive nation-states to develop new-school speech regulation and 
private-infrastructure owners to cooperate with this regulation, whether grudgingly or willingly. 37 


It is usually easier for nation-states to regulate the infrastructure operator or owner than to locate 
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and regulate individual speakers: There may be too many speakers, they may be anonymous or not 
even people, they may be difficult to find, or they may be located outside of the nation-state’s 
jurisdiction. 32 Conversely, infrastructure providers are usually easier to locate, and most have 
good reasons to be receptive to state pressure. 33 They want to make money, and they want to 
expand their markets to reach customers within the nation-state’s jurisdiction. 34 Even if 
infrastructure companies strongly believe in civil liberties and would rather not abridge the speech 
of their customers and end users, they may nevertheless conclude that cooperating with nation- 
states better furthers their profit-making goals. 35 


Fourth, market incentives and repeated public-private interactions have also driven the 
development of private governance and new-school speech regulation. Infrastructure owners’ 
technical capacities for surveillance and control continue to grow over time, not only because of 
market competition and demands from business partners but also as a result of continual political 
pressure from nation-states and the European Union. 36 The more powerful infrastructure 
operators become, and the greater their capacity for governance of large populations of end users, 
the more valuable targets they become for new-school speech regulation. 


The result is a burgeoning dialectic of governing power and public-private cooperation. Private- 
infrastructure companies develop ever greater governing capacities. 37 Nation-states attempt to 
co-opt these capacities through coercion or threats of regulation. This, in turn, causes increased 
development of governing, surveilling, and regulatory capacities. And this, in turn, makes private- 
infrastructure owners even more tempting targets for government pressure—because private 


companies can no longer pretend that they cannot actually do what governments want them to do. 
38 


This dialectic encourages new-school speech regulation, making it ever more important to nation- 
states as a method of surveilling, regulating, and controlling forbidden speech and conduct on the 
internet. This dialectic was not so obvious in the early days of the internet, before the rise of social 
media companies, when surveillance and filtering techniques were far more primitive. But as 
technology companies grew, expanded internationally, and became ever more technically 
proficient, nation-states began to demand more and more from them. 39 


Il. PRIVATE GOVERNANCE AND PRIVATE BUREAUCRACY 


A. PRIVATE GOVERNANCE 


Technology companies’ ever-expanding capacities for private surveillance and control lead naturally 
to viewing them as a new form of private governance. By this | mean that we should think of 
private-infrastructure owners—and especially social media companies—as governing online 
speakers, communities, and populations, rather than thinking of them as merely facilitating or 
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hindering digital communication. 4° Instead of viewing digital-infrastructure companies as mere 


conduits or platforms, we should recognize them as the governors of social spaces. 


Professor Kate Klonick has developed this idea in her study of the emergence of internal 
bureaucracies in social media companies such as Facebook, Twitter, and YouTube. 4’ She explains 
how the concept of community governance, and the creation of large global bureaucracies, 
emerged almost by accident as social media companies sought to enforce their terms-of-service 
agreements and had to respond to pressure from various nation-states to control or curb speech 
that these countries regarded as illegal or undesirable. 42 Faced with an unruly and unpredictable 
collection of all types of people from around the world (not to mention agents of various nation- 
states), these companies learned that they had to govern—that is, promulgate and enforce the 
values and norms that their communities stood for. 4 They did so through a combination of 
contract (that is, terms of service or end-user license agreements) and code. 44 Over time, social 
media companies, which originally thought of themselves only as technology companies, accepted 
their role as community governors and developed elaborate bureaucracies, which are effectively 


governance structures. 4 


The task of governing online spaces need not be wholly public spirited. It may be driven by market 
incentives or by the quest for economic and political power. Facebook has adopted community 
rules because of its business model, which requires that its space be safe, attractive, and absorbing 
for its billions of users around the world. 46 Social media companies cannot afford to scare off their 
customers because they need to capture end users’ scarce attention to make money. The business 
model of social media companies requires vast numbers of individuals to repeatedly check the site, 


read the site, and post to the site so that the company can sell their scarce attention to advertisers. 
47 


Companies like Facebook generate growth—and thus please the demands of their shareholders—in 
one of two ways: First, they can expand their membership to more people around the world. 
Second, they can gain a greater share of their end users’ attention. 48 The first strategy offers 
limited possibilities for a company as large as Facebook; therefore, the second strategy begins to 
dominate. As Professor Tim Wu has pointed out, social media companies have an incentive to make 
their services addictive so that they can garner a larger share of their end users’ attention. 49 


Twentieth-century freedom of speech faced a problem of scarcity of access to media. 5° Twenty- 
first-century freedom of speech faces the problem of scarcity of attention. 5’ The logic of scarcity 
of attention drives the business models of many twenty-first-century digital companies that attract 
end users by offering free (or subsidized) services in exchange for brokering end users’ attention to 


advertisers. 
The capitalist logic of digital media services requires continuous growth either through expansion of 


membership or through expansion of attention. 5? To seize attention, a social media platform must 
have both absorbing content and provide a community in which people feel safe; otherwise end 
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users will not spend time on the site. Hence, the economic logic of advertiser-driven social media 
leads them to become governors of their spaces. 


Moreover, to sell end users’ attention to advertisers, it is necessary to know things about them so 
that advertising dollars are not wasted. The ability to serve different ads to different audiences 
requires knowledge about audiences, and thus the collection of ever-greater amounts of data about 
end users. The logic of digital capitalism, in other words, also drives companies toward surveillance 


as well as governance. 53 


The same logic of digital capitalism that leads to governance and surveillance of end users also leads 
to the creation of bureaucracies, which consist of the company’s digital workers using easy-to- 
apply rules for deciding vast numbers of cases and controversies, while pushing a small number of 
more complicated cases up the chain of decision. 54 This follows naturally from the global nature 
of both the company’s end users and its employees: When both content and employees come from 
everywhere, social media companies need simple, easily understandable, easy-to-apply rules that 
can be followed uniformly. 55 


Another method for lowering costs and ensuring uniformity is to substitute algorithmic for human 
judgment. 56 Algorithmic employees cost even less than human employees: They do not have 
families, they do not take coffee breaks, and they can do some—but by no means all—of the work 
of discovery and selection that human employees can do. Algorithms may be especially useful in ex 
ante blocking of content—for example, identifying child pornography or preventing the upload of 
content that has been digitally watermarked as copyright protected. 57 


Governance by Facebook, Twitter, and YouTube has many aspects of a nineteenth-century 
autocratic state, one that protects basic civil freedoms but responds to public opinion only in 
limited ways. The end users—akin to the citizens or subjects—are in effect unpaid laborers for the 
site, in the same way that anyone who uses open-source software and reports bugs is an unpaid 
laborer for the open-source project. 5® When end users spot bugs, make complaints, or demand 
new features, this helps inform the company, its bureaucrats, and its programmers how best to 
attract and mollify end users and keep profits flowing. Every end user is a potential reporting or 
surveillance device for maintaining community standards. 59 Every time end users complain about 
racist speech or trolling, they are in effect working for Facebook because they provide the company 
with information that helps it enforce its community standards. 6° 


Because the governance of social media companies is generally autocratic, their governance policies 
are, for the most part, nontransparent and waived whenever necessary or convenient. © There is 
normally little in the way of due process for end users, much less a right to a hearing either before or 
immediately after sanctions are applied. 6 Companies often make special exceptions for powerful 
and influential actors and organizations. ©3 But if the speaker is a “puny anonymit[y],” 64 it is far 
more likely that a social media company will sanction or ban the speaker. 65 


B. SHOULD PRIVATE GOVERNANCE BE PRIVATE? 
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Nevertheless, the best alternative to this autocracy is not the imposition of First Amendment 
doctrines by analogy to the public forum or the company town. 66 Of course, new-school speech 
regulation may violate the First Amendment—because the state has passed laws that pressure 
infrastructure providers to do its bidding. 7 But when we focus on social media governance that is 
not the result of new-school speech regulation, our analysis should be different. Social media 
companies should recognize and protect free speech values as well as due process values in the 
resolution of complaints. Even so, it is generally a bad idea to hold social media spaces to the same 


standards as municipal governments under the First Amendment. 


Imposing the same First Amendment doctrines that apply to municipalities to social media 
companies would quickly make these spaces far less valuable to end users, if not wholly 
ungovernable. First Amendment law significantly limits the ability of municipalities to regulate 
anonymous or pseudonymous speech in public forums; 68 yet sites may want to require real names 
or easily identifiable pseudonyms in order to prevent cyberbullying, harassment, and trolling. Under 
current First Amendment doctrine, sites might not be able to ban hate speech or other kinds of 
abusive and emotionally upsetting speech that make the site far less valuable for the vast majority 
of customers. 69 Municipalities can ban fighting words, 7° but speakers on the internet may be 
nowhere near the recipients of their venom so that an immediate breach of the peace is highly 
unlikely. 77 Although the Supreme Court has not declared the tort of intentional infliction of 
emotional distress unconstitutional, it has been careful to suggest that speech that causes 
emotional distress is protected if it discusses matters of public concern. 72 


A final problem is that, unlike municipalities, social media sites cannot levy damages or fines. They 
have only limited sanctions for misbehavior: denying access to the site, either temporarily or perma- 
nently, and removing some or all of an offender’s previous content. 73 Denying access to a public 
forum and removing all of an end user’s content as punishment for previous conduct may create 
problems under prior restraint doctrine. Take the example of defamation: Under current First 
Amendment law, the government—and hence a social media site treated as an arm of the state— 
might not be able to deny access to an end user and remove some or all of the end user’s previous 
posts because he or she had previously defamed a person. Courts might regard this as a prior 
restraint and no more constitutional than denying future access to a public park to a person who 
had previously defamed someone in the park. 74 Moreover, removing all of an end user’s content as 
punishment, even when significant parts of that content constitute protected speech, would seem 
to raise serious First Amendment problems. 


The result is that—at least until the courts begin to treat cyberspaces differently from other public 
fora—applying First Amendment law would cripple social media sites’ abilities to impose civility 
norms. When spaces seem unsafe and are riddled with racist speech and personal abuse, many 
people will avoid them. 


Second, under a First Amendment regime, social media sites would be unable to curate content in 
order to provide personalized feeds. The creation of personalized feeds is inevitably content-based 
—social media sites have to decide what content is likely to be most interesting to their end users. 
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75 As Professor Tarleton Gillespie has pointed out, social media sites thrive on content-based 
moderation, even if the moderation is invisible to most users. 76 The same is true of search 
engines; ideally, their purpose is to help end users reach information that is relevant to their search 
engine queries. Furthering this task requires multiple content-based distinctions about the 


relevance and arrangement of links. 77 


The free speech values characteristic of this environment—and that promise to work best within it 
—do not necessarily conform to First Amendment doctrine. To be sure, the degree of free speech 
protection that exists on these sites is due in no small part to the fact that they were originally 
created by American-led companies and have been deeply influenced by American free speech 
values. 78 But because social media companies operate around the world, they cannot realistically 
apply American First Amendment doctrines everywhere, in part because American free speech law 
requires Americans to tolerate all sorts of things that people in other countries simply will not put 
up with. 79 


Given its global reach, Facebook has decided to have a worldwide policy, which applies more or less 
uniformly in every country. 8° According to this policy, the company takes down hate speech and 
disrespectful speech that would almost certainly be protected by the American First Amendment. 

81 This policy is hardly surprising from a company that seeks to do business around the world. A 
global company that governs a global online community requires global rules on freedom of 
expression that are not necessarily American free speech rules, much less doctrines originally 
designed for public streets and parks. 


C. PRIVATIZED BUREAUCRACY 


So far, this Essay has introduced the ideas of old-school and new-school speech regulation and the 
crucial connection between new-school speech regulation and private governance. The last piece 
of the puzzle emerges out of this connection. It is the emerging system of privatized bureaucracy. 82 


For the reasons described above, new-school speech regulation depends on the expansion and 
promulgation of private governance. 83 Indeed, new-school speech regulation and private 
governance egg each other on. As digital-infrastructure companies become increasingly powerful in 
governing their spaces and collecting and analyzing content from their end users, nation-states may 
demand more from them through new-school speech regulation. As nation-states attempt to co- 
opt and coerce private-infrastructure operators, they increasingly attempt to get private companies 
to take on state functions of speech regulation and surveillance. As social media and search engine 
companies develop governing bureaucracies and algorithms, nation-states seek to harness that 
capacity to accomplish the nation-state’s governance goals. These processes lead to a new 
phenomenon: privatized bureaucracy. Bureaucracies within private-infrastructure companies 
(including their contractors) serve as the front line for the nation-state’s governance of online 
speech and conduct. 
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Two examples demonstrate how this phenomenon works: the right to be forgotten, which applies in 
the European Union generally, and Germany’s recent NetzDG law. 84 


Consider how the European Union protects the right to be forgotten. Suppose that a petitioner 
objects to the presence of an embarrassing article on a search engine such as Google. The European 
Court of Justice (ECJ) has ordered Google to develop a bureaucratic system for deciding in the first 
instance whether a particular article should be delinked from its search engines. If the petitioner 
disagrees with Google’s decision, he or she can sue in the courts. 85 This is, in essence, a system of 
administrative law, requiring an exhaustion of administrative remedies before one can use the court 


system. But the administrative agency in this case is a private company. 


The ECJ chose this solution because the European Union and its member states lack the technical 
capacity to monitor the internet and protect the right to be forgotten on their own. 86 The number 
of complaints is likely to be very large and processing these complaints would require the creation 
of a sizeable new bureaucracy in each member state. 87 In order to protect those rights that the 
ECJ asserts should exist under European law, the European Union has essentially deputized a 
private company to serve as its bureaucracy. 88 This deputizing of privately owned infrastructure 
companies is the culmination of the logic of new-school speech regulation. 89 


Similarly, Germany’s new NetzDG law was designed to co-opt social media companies into 
monitoring and taking down prohibited content in Germany, including hate speech. 9° Although 
some internet companies, such as Facebook, already have hate speech policies, Germany 
demanded stricter enforcement and prompt takedown—essentially within 24 hours of notice for 


|” 


“manifestly unlawful” speech. 9’ Failure to comply with the state’s requirements leads to sanctions 


against the company. 97 


From one perspective, NetzDG is just collateral censorship—a nation-state puts pressure on 
digital-infrastructure companies to block, take down, and censor content by end users. But from 
another perspective, NetzDG involves an agreement between the German state and various private 
companies in which the companies act as a private bureaucracy that implements the state’s speech 
policies. Because Germany currently lacks the technical capability to perform this task on its own, it 
coerces or co-opts Facebook, Google, and Twitter to do it instead. Once again, this is the logical 
outcome of new-school speech regulation. 


One might make four different objections to a government program like NetzDG, and it is 
important to distinguish them because they represent four different objections to new-school 
speech regulation. Three of these concern speech, while the last concerns surveillance. 


First, one might object to Germany’s substantive hate speech doctrines as insufficiently speech 


protective. This objection is really not about internet regulation at all, for Germany would 
presumably enforce the same restrictions on speech that did not appear on social media sites. 
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Second, one might object that Germany will attempt to impose its content regulation outside of its 
geographical boundaries. Because German citizens may access the internet everywhere (or use 
internet proxies to simulate being outside the country), Germany may eventually demand global 
jurisdiction for its speech regulations. Similarly, the European Union and its member states may try 
to enforce the right to be forgotten around the world. 93 


In some cases, it may not even be necessary to formally assert jurisdiction over a company’s 
operations around the world. Governments can achieve similar effects through indirection. They 
may encourage companies to alter their terms of service to better conform to the state’s 
substantive law. 94 Then governments and associated law enforcement agencies can inform 
companies that content violates the company’s terms of service and request removal, achieving 
worldwide enforcement by other means. 95 


This objection is distinct from the question of whether Germany or the European Union have 
adopted the correct substantive understanding of the right of freedom of speech. Instead, this 
objection is related to new-school speech regulation because enforcing universal jurisdiction 
involves coercing or co-opting infrastructure providers to enforce particular speech norms 
universally. The better infrastructure providers are at locating and enforcing speech regulations 
around the world, the more nation-states may be tempted to harness these technical capacities for 


their own ends. 


Third, quite apart from concerns about substantive law and global jurisdiction, speakers get no 
judicial determination of whether their speech is legally protected or unprotected when private 
companies block, censor, or take down their speech. Instead, some nontransparent form of private 
governance or bureaucracy serves as prosecutor, judge, jury, and executioner. Speakers thus get no 
due process protections of notice or hearing. This is a problem of collateral censorship, which, as 
noted above in section I.A, has aspects of administrative prior restraint. 


Fourth, when nation-states co-opt private infrastructure to regulate speech, they may also co-opt 
private infrastructures’ capacities for surveillance, data collection, and analysis to solve their own 


problems of governance and control. 96 


One must pay attention to all four of these issues when considering any question of online speech 
today. The first issue—the question of substantive standards—is often the most salient to 
Americans because Americans generally have a much more libertarian free speech policy than the 
rest of the world. The second is the looming possibility of global jurisdiction, with countries vying 
with each other to impose their parochial speech regulations on the entire world—a race to the top, 
which, in American eyes, would be a race to the bottom. The third and fourth problems—privatized 
bureaucracy and surveillance—arise from the combination of new-school speech regulation and 


ever more technically effective private governance. 


The result is a complicated array of relationships of power, control, and surveillance. End users, 
citizens, legacy media, and civil-society organizations are now targets of both old-school and new- 
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school speech regulation by nation-states, as well as the subjects of private governance by digital- 


infrastructure companies. 


End users are by no means powerless in this new environment—for example, coordinated 
campaigns by end users and mass media may pressure companies to change their policies. 97 The 
larger point, however, is that speakers face multiple threats from public and private governance and 
power, instead of merely the traditional threats of old-school speech regulation. 


HI. PROTECTING FREE SPEECH VALUES IN A PLURALIST 
SYSTEM OF REGULATION 


If the characteristic feature of free speech regulation in our time is a triangle that combines new- 
school speech regulation with private governance, then the best way to protect free speech values 
today is to combat and compensate for that triangle’s evolving logic of public and private regulation. 
The solution will not necessarily—or even primarily—involve enforcing the doctrines of the First 
Amendment jot for jot against private-infrastructure providers. To be sure, it will concern the free 
speech values that animate the First Amendment. But the best way to protect those values is not to 
apply doctrines developed for states as rules for private actors. Instead, protecting free speech ina 
digital age often involves technical, regulatory, and administrative solutions that apply in contexts 
where the First Amendment does not reach. 98 Judge-made doctrines of First Amendment law can 
do only limited work, and sometimes they will actually hinder necessary reforms. 99 


Protecting free speech values in a pluralist model has two basic goals. The first goal is to prevent or 
ameliorate, as much as possible, collateral censorship and new forms of digital prior restraint. The 
second goal is to protect people from new methods of digital surveillance and manipulation—new 
methods that emerged from the rise of large multinational companies that depend on the 
collection, surveillance, analysis, control, and distribution of personal data. 10° 


The four problems mentioned above—collateral censorship, privatized prior restraint, surveillance, 
and manipulation—are predictable features of private governance, of new-school speech 
regulation, and of public-private cooperation and co-optation. To protect free speech values in the 


new era, one must aim at all of them. 


A. PERMISSIBLE GOVERNMENT REGULATIONS: STRUCTURAL REFORM AND 
PROCOMPETITION POLICIES 


Protecting free speech values does not mean rejecting all government regulation of digital 
infrastructure. Some regulations do not produce the problems of new-school speech regulation— 
collateral censorship and digital prior restraint. 1°’ To the contrary, they actually protect free 
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speech values. These kinds of regulations may not only be perfectly appropriate, they may actually 
be necessary to provide practical freedom for end users. 


One example of permissible regulation is the structural regulation of telecommunications facilities. 
Examples include municipal broadband projects; open-access rules, which make it possible to have 
many different kinds of internet service providers; 1°? and network neutrality rules, which prevent 
discrimination against content and applications. 1°3 These structural regulations of infrastructure 
are not new-school speech regulation: They do not encourage collateral censorship or digital prior 
restraints, and they do not raise the same problems of due process. In fact, network neutrality helps 
avoid many of the problems of collateral censorship because broadband providers may not block or 


slow down traffic based on its content or viewpoint in a network neutrality regime. 1°4 


A second and very important kind of regulation is procompetition regulation, which might include 
both antitrust law and media concentration rules. 195 Procompetition policies are important not 
only because of the potential power of privately owned bureaucracies but also because of their 
potential vulnerabilities. 16 The hacking of the 2016 election might have been different, and 
possibly less effective, if there were multiple Facebooks with different affordances, technologies, 
and advertising models. 1°7 There is only one Facebook today not simply because of economies of 
scale and network effects but because Facebook strategically bought up a number of potential 
competitors, incorporating some of their innovations and blocking others. 8 In this way, it 
forestalled the development of a wide range of potential competitors and innovators in social 


media. 


More social media competitors, each with differing approaches and goals, would provide more 
platforms for innovation, more software features, more types of security measures that hackers 
would have to circumvent, more models for social spaces and communities, and a wider variety of 


speech policies. 1°9 


With stronger enforcement of antitrust and procompetition laws, innovations might have 
proliferated more widely, and we might have a healthier competition among social media companies 
and their sorting algorithms. Although we cannot be certain that this would have made it harder for 
foreign propaganda and fake news to proliferate and disrupt our democracy, it is generally harder to 
attack and compromise twelve targets than to attack and compromise one. 


One might object that this degree of fragmentation—we might even call it balkanization—is 
undesirable. "° But procompetition policies serve democratic values in a second way. Modern 
democracies increasingly rely on social media to facilitate public conversation, organize public 
discussion, and enforce civility norms. ™ Therefore, it is especially important to make sure that 
there are many such organizations, in order to prevent a small number of powerful for-profit 
companies from dominating how public opinion is organized and governed. Social media 
enforcement of civility norms is imperfect and often arbitrary, "2? and some organizations, like 
Facebook, attempt to impose the same standards around the world. "3 Thus, when people expect 
and even demand that a multinational corporation like Facebook ban hate speech, it is important to 
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have many Facebooks, not a single one. The flip side of the expectation that social media sites 
should enforce civility norms is the need for multiple social media sites serving different values and 
different publics. 


One might also object that network effects will prevent broad diversity in social media because 
users will flock to the platforms with the largest user base. 4 Yet network effects will not 
necessarily prevent the growth of multiple social media sites. "5 


First, Facebook, like MySpace before it, will not be dominant forever. Often people—and especially 
generations—migrate from application to application without completely abandoning any of them. 

16 How much time people spend on different sites may be fluid and may change over time as 
people age or have new experiences; moreover, the sites themselves may add new features as they 
compete for scarce attention. 


Second, Professor Klonick has pointed out that people may see social media sites like Facebook, 
Twitter, and YouTube as complementary goods rather than rival goods. "7 People might use all 
three services for different purposes. 


Third, social media sites are not like countries—one can both inhabit and be a “citizen” of many of 
them at the same time. The best model for the new digital public sphere is not the familiar model of 
competition between geographically distinct states. "8 Rather, it is one of diaspora, in which 
immigrants have connections both to their country of origin and to their current country and may 
also have relatives in many different countries. Digital diaspora may be a better model for thinking 
about the ecology of social media than the model of exclusionary network effects. 


B. THE RESPONSIBILITIES OF PRIVATE INFRASTRUCTURE 


Different parts of the internet infrastructure should have different responsibilities to protect 
freedom of speech online. For convenience, one might divide the digital infrastructure of free 
expression into three basic groups. "9 


The first group is basic internet services. It consists of four types of companies: 


(a) Hosting services (such as Amazon Web Services or Gmail), which host websites, software 
applications, and platforms; 


(b) Telecommunications services, which include internet backbone operators, ISPs, transit 
providers, and certificate authorities (which issue SSL certificates to websites and other 
applications); 


(c) Domain name services, which include registrars that register domain names (such as 
GoDaddy), registries that run top-level domains (such as Verisign), and DNS providers that resolve 


domain names (such as Cloudflare and Google); and 
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(d) Caching and defense services (such as Akamai and Cloudflare), which smooth and speed up 
internet traffic and may also provide cybersecurity and defense against DNS attacks. 12° 


The second group consists of payment services that allow people to conduct business and make 
payments online (such as Mastercard, Visa, and PayPal). Although payment services do not 
regulate traffic flows, many online enterprises would be effectively impossible without them. 171 


The third group consists of content curators. These companies include both platforms (such as 
Facebook, YouTube, Twitter, and Instagram) and search engines (such as Google). These 
companies make regular and pervasive content-based decisions as part of their business models 
using human bureaucracies, algorithms, or some combination of the two. 122 


Generally speaking, basic internet services should adopt policies of nondiscrimination with respect 
to the content and viewpoint of traffic that flows through them or that is stored on them. Network 
neutrality rules attempt to enforce this principle against broadband providers and ISPs, but 
analogous principles should apply to the rest of the delivery system. 


For example, caching and defense services like Cloudflare should not, as a general rule, discriminate 
on the basis of content or viewpoint. There are three reasons for this. First, as a practical matter, 
these services are increasingly important for unpopular speakers or for speakers likely to be targeted 
by others. '23 Second, withholding caching or defense services for particular disfavored sites will 
likely have significant collateral effects for other content on those sites that deserves protection. 
Third, withholding services will have many features of an administrative prior restraint: The 
decisions will be nontransparent and lack due process. 


The analysis of DNS services is a little different. The initial grant of a domain name is usually 
content-based, if only because two applicants cannot have the same domain name; moreover, 
permissible top-level domain names are regulated by the Internet Corporation for Assigned Names 
and Numbers (ICANN). 774 The point, rather, is that once a domain name has been granted, the 
DNS system should not refuse to resolve a domain name because DNS service providers 
disapprove of the content appearing on a site that employs a given domain name. A fortiori, gov- 
ernments should not try to leverage the domain name system to block or censor content. 125 
Suspension of domain names, refusal to resolve domain names, and blocking content by domain 
name offer extreme examples of collateral censorship. '26 As before, these decisions will also likely 
lack transparency, notice, and due process. 


The second group, payment systems, presents still another set of problems. We might best 
analogize payment systems to public accommodations. They should be open to all people, groups, 
and businesses that do not use the service to engage in illegal activities. Public accommodations 
usually protect people against discrimination based on their identities—race, religion, sex, and so 
forth. In this context, however, the goal is to prevent discrimination based on the content of what 
people lawfully publish online. Even so, payment systems should be able to refuse to do business 
with those who seek to use their systems to facilitate illegal enterprises, which may include the sale 
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of content whose distribution is illegal in a particular jurisdiction. But where the publication of 
content is not illegal, payment systems should not discriminate among their customers. In 2011, for 
example, the United States put pressure on payment systems to refuse to do business with 
WikiLeaks. 127 What made this episode especially worrisome was that, at that point, WikiLeaks was 
in much the same position as an American newspaper, which could not be prosecuted for 
publishing the very same information. 128 


Because basic internet services and payment systems should not engage in content regulation— 
with certain exceptions for the DNS system noted above—government regulation that enforces 
nondiscrimination obligations should ordinarily not be objectionable on free speech grounds. These 
companies should not exercise editorial control in the first place; hence government regulations 
that enforce obligations similar or analogous to common carriage, nondiscrimination, or public 
accommodation should normally be appropriate, both from a First Amendment and a more general 
free speech perspective. 129 


The problem, of course, is that nation-states may be tempted to do precisely the opposite—not to 
forestall content discrimination but to demand it through new-school speech regulation. Once the 
telecommunications system, the DNS system, and the system of electronic payments begin 
blocking, censoring, or discriminating against certain speakers, nation-states will attempt to 
piggyback on their technical capabilities. As we have seen, state pressure on infrastructure owners 
to surveil, block, and filter content creates predictable problems of collateral censorship and 
privatized prior restraint. 


IV. THE OBLIGATIONS OF DIGITAL CURATORS—CURATIONAL 
DUE PROCESS AND INFORMATION FIDUCIARIES 


Curators are in a different position than either payment systems or basic internet services because 
they curate and personalize information for end users. They also facilitate communication through 
curation. For example, an end user’s Facebook feed does not offer every possible posting from the 
user’s Facebook friends in the order they were posted; instead, Facebook decides which posts are 
most relevant and in what order to display them. 13° Google tries to provide the links that are most 
helpful to end users who make search requests, '3’ and it tries to deter companies that seek to 
game the system of search engine results. 132 


Because companies like Facebook and Google act as curators and personalizers, they cannot really 
avoid making decisions about content. We should therefore think about their obligations differently 
than payment systems and basic internet services. Familiar concepts like content and viewpoint 
neutrality are simply unhelpful in describing their responsibilities in the emerging global system of 
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free expression. Above all, these curators need to be trustworthy providers of search and 


communications services and nonarbitrary in their governance of communities. 


A. DIGITAL CURATORS AS PROFESSIONALS AND THE SUCCESSORS OF TWENTIETH- 
CENTURY MASS MEDIA 


The obligation of trustworthiness makes digital curators both similar to and different from 
twentieth-century (or “legacy”) mass media such as newspapers, broadcasters, and cable channels. 
Like twentieth-century mass media, digital curators have become important custodians of the 
public sphere and of democratic self-government. Hence, whether they like it or not, digital 
curators have social and moral obligations to the public—as opposed to legal obligations. With 
those obligations comes a corresponding duty to develop and abide by professional norms of 
curation and governance. 


Social media companies and search engines have social and moral obligations to the public because 
they perform three connected public services. First, they facilitate public participation in art, politics, 
and culture. Second, they organize public conversation so that people can easily find and 
communicate with each other. Third, they curate public opinion by providing individualized feeds 
and search results, and by enforcing civility norms through their terms-of-service obligations and 
community guidelines. 


Social media companies and search engines present themselves as more than ordinary profit- 
making enterprises. '33 They explain that they use their special technological expertise to promote 
public-spirited goals like access to knowledge, freedom of expression, and community building. 134 
In this way, they encourage the idea that they do act and should act according to public-regarding, 
professional norms. Moreover, social media companies and search engines invoke these 
professional and public-regarding norms to justify their decisions to organize search-engine results, 
to curate public discourse, and to enforce (or sometimes refrain from enforcing) civility norms. 135 
The public, politicians, and civil-society organizations repeatedly push back, claiming that digital 
companies have fallen short of these professions of public-spiritedness and demanding that 
companies act according to the public interest. 136 


A similar process happened to journalism in the early twentieth century. Newspapers were 
confronted with the rise of propaganda, advertising, and public relations. Seeking to differentiate 
themselves from these practices, newspapers gradually accepted that they had distinctive 
professional obligations to the public in how they covered and reported the news. 737 This growing 
sense of responsibility to the public developed into what we now know as the professional norms of 
modern journalism. 138 The twentieth-century vision of objective journalism in the public interest 
did not arise overnight—it was shaped by economic, social, and technological developments. 


Just as in the case of twentieth-century mass media, however, the state constitutionally cannot 
force these professional norms—or their twenty-first-century equivalents—on digital curators. But 
this does not mean that the public cannot or should not demand these norms. We are beginning to 
see a slow and halting evolution of platforms’ self-understanding precisely along these lines. This 
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learning process is the result of wave after wave of public pressure on companies like Google, 
Facebook, and Twitter, often facilitated by journalists who themselves apply professional norms of 
news reporting developed in the previous century. 139 Companies that once viewed themselves 
purely as technology companies are beginning to understand their public responsibilities as twenty- 


first-century media companies. 


In saying this, it is important not to wax nostalgic over twentieth-century mass media or to assume 
that the twentieth century represents a lost, golden age of media responsibility. 14° Nor should we 
assume that twenty-first-century media will adopt professional norms identical to those of 
twentieth-century journalism. The point is rather that twentieth-century media, with all of its 
faults, served as a countervailing force to government power in a democracy. In the same way, 
twenty-first-century media companies, at best, may provide platforms for democratic organization 
and protest and act as checks on the power of territorial governments, even as these governments 
are necessary checks on technology companies’ burgeoning economic and political power. 


B. LEGAL OBLIGATIONS—CURATIONAL DUE PROCESS 


If digital curators were just like twentieth-century mass media, that would be the end of the story. 
These companies would have public obligations—that is, moral duties—to develop professional 
norms in the public interest, and the public (and legacy media) might pressure them into adopting 
and adhering to those norms. The state, however, would be forbidden from enforcing these norms 
by law. But that is not the end of the story. The new digital curators differ from twentieth-century 


mass media in two important respects. 


The first difference is that digital media companies have curatorial obligations of due process. 
These obligations made little sense in a world in which very few people had access to mass media 
but are central in a world in which everyone is a broadcaster. For this reason, to the extent that 
digital curators block, censor, or take down content from their end users, they have obligations of 
due process toward their end users. 1# That is especially so if their content regulation is at the 
behest of nation-states employing new-school speech regulation. 


To see how these due process obligations might operate in practice, a good place to start is the 
Manila Principles on Intermediary Liability, a series of reform proposals developed by civil society 
organizations in 2015. 12 The Manila Principles require, among other things: (1) clear and public 
notice of the content-regulation policies companies actually employ; (2) an explanation and an 
effective right to be heard before content is removed; and (3) when this is impractical, an obligation 
to provide a post facto explanation and review of a decision to remove content as soon as 
practically possible. 13 One might call these and similar norms the obligations of curational due 
process. 


The Manila Principles focus on removing content that is illegal in a given country, 44 but the same 
principles could also apply to content that violates the company’s terms of service or end-user 


license agreement. In fact, instead of passing new speech regulations, nation-states may find it 
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more convenient to press curators to enforce their existing terms of service. This has the additional 
advantage of leveraging private speech codes that operate globally to serve the nation-state’s 
parochial ends. 145 


Curational due process made little sense for twentieth-century mass media because twentieth- 
century mass media largely published their own content or the content of a relatively small number 
of people and businesses. Most people had no access to mass-media publication, and few people 
expected an explanation or a right to be heard either before or after the New York Times rejected 
their proposed letters to the editor. Twenty-first-century media companies, by contrast, primarily 
publish content by the general public. Digital curators exist to facilitate mass cultural participation, 
and their end users expect and depend on the fact that curators will help them in this process. 
Therefore, curators need to provide assurances that when they block or limit participation, they are 
not being overbroad or arbitrary. 


Legislation that requires digital curators to provide due process would not necessarily violate the 
First Amendment. Curators would probably argue that such rules would interfere with their editorial 
functions. 146 But one can avoid constitutional problems by making due process obligations part of 
a safe harbor from intermediary liability. 


Section 230 of the 1996 Telecommunications Act protects digital curators from liability for content 
appearing on their sites. '47 Some aspects of intermediary immunity are probably required by the 
Constitution, so that if Congress repealed § 230, certain constitutional protections would still be in 
force. 48 For example, it might be unconstitutional to hold digital curators strictly liable for any 
defamatory or obscene content that appears on their sites. 149 But the boundaries of constitutional 
protection are uncertain. Would a negligence standard be sufficient? What about other kinds of 
unlawful content? 15° What if digital curators are notified that the material is defamatory, tortious, 
or otherwise unlawful? 15’ These and similar questions remain unsettled. Moreover, § 230(c)(2)— 
which holds digital curators harmless for editing or deleting content of end users—is probably not 
required by the First Amendment. 152 


The best way to guarantee curatorial due process, therefore, is to resolve these uncertainties by 
creating a safe harbor provision that would amend § 230. '53 If digital curators agree to adopt 
something similar to the Manila Principles, they will retain their intermediary immunity. If not, they 
will have to be content with constitutional limitations on intermediary liability, which are both 
uncertain and likely to be far less than the statutory guarantee. 


Note that this solution would employ a new-school technique because it would reduce the 
intermediary immunity of digital curators with respect to the status quo. But it would use that 


technique for the opposite goal of most new-school speech regulation: It would attempt to protect 
the free speech interests of end users. 


C. LEGAL OBLIGATIONS—INFORMATION FIDUCIARIES 
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The second difference between twentieth-century mass-media companies and twenty-first- 
century digital curators is that twenty-first-century companies have developed elaborate 
technologies and techniques for individualized surveillance, manipulation, and control that were 
not really possible for twentieth-century mass media. 154 To be sure, twentieth-century mass 
media also hoped to appeal to certain demographics in order to attract advertisers. Yet their 
abilities to surveil, target, manipulate, and even addict their audiences could not be so effective or 
so precise as those of twenty-first-century companies. 


Indeed, the characteristic feature of twenty-first-century digital media companies is not merely 
that they enable mass participation. It is that in doing so they also engage in mass data collection 
and surveillance, and that they develop ever more effective means for influencing (and thus 
potentially manipulating) their audiences in order to gain their scarce attention. 155 The flip side of 
mass cultural participation is mass personal surveillance, and the danger of widespread digital parti- 
cipation is widespread digital manipulation. 


Digital curation is not simply the selection of content for end users; it also involves using knowledge 
about end users to control, shape, and govern their behavior. 5° Digital curators are private 
governors not only in establishing and enforcing community norms but also in their attempts to 
govern and direct end users through surveillance. The problem of digital curators, which makes 
them different in kind from twentieth-century mass-media companies, is the far greater danger 
that they will engage in acts of manipulation and breach of trust through the use of personal data. 


In the algorithmic age, many digital companies—and not merely digital curators—take on new kinds 
of obligations. These new obligations arise from people’s increasing dependence on and 
vulnerability to digital services that collect data about them but whose operations are not 
transparent to them. Companies that create and maintain these relations of digital dependence and 
vulnerability should be considered information fiduciaries toward their end users. 157 


We rely on digital companies to perform many different tasks for us. In the process, these 
companies learn a great deal about us, but we do not know very much about their operations. 158 
As a result, we are especially vulnerable to them, and we have to trust that they will not betray us or 


manipulate us for their own ends. 


The law has long recognized that clients or patients of professionals like doctors and lawyers are ina 
similar situation: We need to trust these professionals with sensitive personal informational about 
ourselves, but they could injure us as a result. Therefore the law treats them as fiduciaries. 159 
Fiduciary relationships are relationships of good faith and loyalty toward people who are in special 
positions of vulnerability. Fiduciaries have special duties of care, confidentiality, and loyalty toward 
their clients and beneficiaries. 16° 


Because digital companies collect enormous amounts of data about their end users, and use this 
data to predict and control what end users will do—including, among other things, matching them 
with advertisers—digital curators are perhaps the most important example of the new information 
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fiduciaries of the digital age. ©’ Even so, we should treat the analogy to doctors and lawyers with 
some care. The kinds of fiduciary duties that a company has depend on the nature of its social role 
and its business. 1°? Digital companies are not trained professionals like doctors and lawyers. They 
offer a different set of services, consumers expect different things from them, and therefore we 
should expect that they will not have all of the same obligations as doctors and lawyers. 163 


For example, unlike doctors and lawyers, social media companies and search engines offer their 
services for free in return for the right to serve targeted ads to their end users. The practice of 
offering free or heavily subsidized services in return for surveillance and collection of data creates a 
potential conflict of interest between end users and digital companies. Companies will always be 
tempted to use the data in ways that sacrifice the interests of their end users to the company’s 
economic or political interests. Nevertheless, unless governments outlaw the practice of financing 
free (or subsidized) digital products altogether, one must start with the assumption that the law can 
cure potential conflicts of interest through appropriate regulation; if so, this means that social 
media companies will be able to monetize personal data in some ways but not in others. Their 
fiduciary duties will constrain the ways they are allowed to collect, monetize, and employ end-user 
data. What constitutes a breach of trust depends on the nature of their business, and this, in turn, 
depends on what consumers would reasonably consider unexpected or abusive for digital 
companies to do. 14 


A good example of how information fiduciaries should not act—and how fiduciary duties would 
constrain their behavior—is the story of how Facebook allowed third parties to exploit its end users’ 
data. This practice came to light in the Cambridge Analytica scandal in the spring of 2018. 165 


Until the middle of 2014, Facebook had a policy of sharing access to end-user data with third 
parties, including for-profit companies and academic researchers. © This practice offered 
Facebook additional ways to monetize consumer data. '7 In 2014, Facebook allowed a data 
scientist, Aleksandr Kogan, to conduct social-science experiments using end-user data. 168 Kogan 
used Amazon’s Mechanical Turk and similar platforms to find people who were willing to take a 
personality quiz for a few dollars; the participants signed onto the test using their Facebook 
accounts. 169 This gave Kogan access to the data that Facebook associated with their personal 
accounts as well as the data of all of their Facebook friends. 177° In this way, Kogan was able to 
leverage the approximately 300,000 users who took the quiz to obtain access to some 87 million 
end users’ data profiles. 177 


What Kogan did not tell Facebook, however, was that he was secretly working with Cambridge 
Analytica, a for-profit consulting company that uses personal data to serve targeted political ads 
based on psychological profiles constructed from the data. 172 Kogan violated Facebook’s platform 
policy for researchers and scientists by turning over the data to a for-profit company. 173 Facebook 
learned about the arrangement in 2015 but did not reveal it to the public. '74 It asked Kogan and 
Cambridge Analytica to delete the data they had harvested but did not ensure that the data was 
actually erased, and Cambridge Analytica kept the data. '75 When the news of the arrangement 
leaked out in the spring of 2018, it caused a scandal, and Facebook’s founder, Mark Zuckerberg, was 


https://columbialawreview.org/content/free-speech-is-a-triangle/ 24/27 


06/11/2023, 10:47 FREE SPEECH IS A TRIANGLE - Columbia Law Review 
asked to testify before Congress. 176 Following the disclosures, Zuckerberg admitted that the 
company had “made mistakes” and described the scandal as a “breach of trust” toward his end 
users. '77 That well describes the central issue. 


As an information fiduciary, Facebook has three different kinds of duties toward its end users: a 
duty of confidentiality, a duty of care, and a duty of loyalty. 1778 The duties of confidentiality and 
care mean that Facebook must keep its customers’ data confidential and secure. It must make sure 
that fiduciary duties “run with the data”: In other words, Facebook must ensure that anyone who 
shares or uses the data is equally trustworthy and is legally bound by the same legal requirements of 
confidentiality, care, and loyalty as Facebook is. The company must vet its potential partners to 
make sure that they are ethical and reliable, subject them to regular audits, and, if they violate the 
terms of their agreements, it must take steps to get back all the data that the company shared with 
them. 


Facebook failed these duties in several ways. It did not properly limit who could use its data and for 
what purposes, it did not vet or audit its partners properly, and it did not claw back the data 
obtained in violation of its policies. In short, it breached its duties of confidentiality and care 
because it did not keep its data confidential and secure. 


Next consider the duty of loyalty. The previous discussion assumed that Facebook would not 
breach its duty of loyalty simply by serving targeted ads for consumer products in return for free 
services. In part, that is because people more or less expect that Facebook will serve them ads 
based on the data it collects. But when Facebook departs from these consumer expectations to 
benefit itself to the disadvantage of its end users, it may breach its duty of loyalty. The problem 
arises when Facebook uses the data in unexpected ways that people would find offensive and a 
breach of trust. The problem is exacerbated when Facebook shares data with third parties without 
adequate controls over use and disclosure, or when Facebook allows third parties access to its end 
users by having end users sign in to third-party applications through their Facebook accounts. It is 
one thing for Facebook to serve you ads for shampoo; it is quite another for Facebook to hand your 
data off to third parties who have no qualms about manipulating you. 


In fact, the Cambridge Analytica scandal appears to have been only the tip of the iceberg. In the 
hopes of increasing profit margins, Facebook granted access to end users’ information to a host of 
for-profit companies without adequate safeguards as to whether companies were manipulating its 
end users. 179 This created a conflict of interest because it gave Facebook an incentive to look the 
other way, which it apparently did. 18° Facebook may have also breached its duty of loyalty by 
allowing third parties to perform social-science experiments on its end users without the equivalent 
of a human-subjects review board to minimize harm and to prevent overreaching and manipulation. 

781 Finally, if, as critics charge, Facebook designed its applications and employed its end users’ data 
to psychologically manipulate and addict them to the site, 182 it would also have breached its duty 
of loyalty, creating a conflict of interest between the company and its end users. 
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The duties of information fiduciaries depend in part on what is reasonable to expect from them 
given their business models. But the most general obligation of digital-information fiduciaries is that 
they may not act like con artists. 183 They may not induce trust in their end users to obtain personal 
information from them and then turn around and betray that trust by harming and manipulating 
them for the company’s own benefit. Digital businesses may not hold themselves out as providing 
safe and welcoming digital communities that respect privacy and then manipulate their end users; 
nor should they be permitted to give access to end-user data to third parties who will not accept 
similar duties of care, confidentiality, and good faith. 184 


Although companies can violate these duties when they violate their privacy policies, fiduciary 
duties extend beyond the precise terms of those privacy policies to duties of good faith, respect, 
and nonmanipulation. 85 Social media companies engage in manipulation when—under 
conditions of extreme information asymmetry and vulnerability—end users must provide 
information about themselves in order to use the service, and companies use this information in 
ways that both benefit the fiduciary and harm the end user. Governments may act to protect these 
obligations of good faith, respect, and nonmanipulation, which sound both in consumer protection 
and privacy. 


Digital curators operating in the United States may object to any regulation of their operations on 
the ground that the First Amendment protects their right to collect, collate, analyze, use, and 
distribute data as they choose. Although the Supreme Court has suggested that data is speech, 186 
the protection of fiduciary relationships between social media companies and their end users 
should be constitutional for two reasons. 


First, information gathered by digital curators in the context of a fiduciary relationship is not part of 
public discourse any more than the information gathered in the course of other fiduciary 
relationships like those between clients and doctors, lawyers, and estate managers. 187 The First 
Amendment allows governments to regulate fiduciaries’ collection, collation, use, and distribution 
of personal information to prevent overreaching and breach of trust. 188 In the same way, the First 
Amendment should not foreclose regulations designed to protect the relationships of trust between 
the new class of digital-information fiduciaries and their end users. 


Second, Congress can avoid any potential constitutional difficulties under the First Amendment by 
creating safe harbors for digital companies as described above. 189 Professor Jonathan Zittrain and 
| have proposed a Digital Millennium Privacy Act under which the federal government would 
preempt state regulation if digital media companies accept the obligations of information 
fiduciaries toward their end users. 19° Offering digital media companies greater protections than 
the Constitution affords as part of a grand bargain to protect end users should be constitutional. 


CONCLUSION 
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Free speech today is a triangle. Its three corners are nation-states, private infrastructure, and 
speakers. 


This triangle creates three problems: (1) new-school speech regulation that produces collateral 
censorship and digital prior restraint; (2) abuse by privatized bureaucracies that govern end users 
arbitrarily and without due process and transparency; and (3) digital surveillance that facilitates 


manipulation. 
Three reforms will help address these problems: (1) structural regulation that promotes 
competition and prevents discrimination by payment systems and basic internet services; (2) 


guarantees of curatorial due process; and (3) the recognition of a new class of information 
fiduciaries with duties of trustworthiness and good faith toward their end users. 


Archived Announcements The Bluebook Archived Issues 


435 West 116th Street New York, NY 10027 Y in 
tel: (212) 854-4398 


Copyright © 2023 Columbia Law Review 


https://columbialawreview.org/content/free-speech-is-a-triangle/ 27/27 


